Method and system for selectively breaking prediction in video coding

ABSTRACT

Described are techniques in video coding and/or decoding that allow for selectively breaking prediction and/or in loop filtering across segment boundaries between different segments of a video picture. A high layer syntax element, such as a parameter set or a slice header, may contain one or more indications signalling to an encoder and/or decoder whether an associated prediction or loop filtering tool may be applied across the segment boundary. In response to such one or more indications, the encoder and/or decoder may then control the prediction or loop filtering tool accordingly.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/192,079, filed Jun. 24, 2016, which is a continuation of U.S. patentapplication Ser. No. 15/008,573, filed Jan. 28, 2016 (now U.S. Pat. No.9,794,573), which is a continuation of U.S. patent application Ser. No.14/699,246, filed Apr. 29, 2015 (now U.S. Pat. No. 9,313,505), which isa continuation of U.S. patent application Ser. No. 13/336,475, filedDec. 23, 2011 (now U.S. Pat. No. 9,060,174), which claims priority toU.S. Provisional Patent Application No. 61/427,569, filed Dec. 28, 2010.The entire contents of each of the foregoing applications isincorporated herein by reference.

FIELD

Embodiments of the invention relate to video compression, and morespecifically, to the selective use of prediction and in-loop filteringmechanisms at picture segment boundaries of video pictures.

BACKGROUND

Digital video capabilities can be incorporated into a wide range ofdevices, including digital televisions, digital direct broadcastsystems, wireless broadcast systems, personal digital assistants (PDAs),laptop or desktop computers, video cameras, digital recording devices,video gaming devices, video game consoles, cellular or satellite radiotelephones, and the like. Digital video devices may implement videocompression techniques, such as those described in standards likeMPEG-2, MPEG-4, both available from the International Organization forStandardization (“ISO”) 1, ch. de la Voie-Creuse, Case postale 56,CH-1211 Geneva 20, Switzerland, or www.iso.org, or ITU-T H.264/MPEG-4,Part 10, Advanced Video Coding (“AVC”), available from the InternationalTelecommunication Union (“ITU”), Place de Nations, CH-1211 Geneva 20,Switzerland or www.itu.int, each of which is incorporated herein byreference in their entirety, or according to other standard ornon-standard specifications, to encode and/or decode digital videoinformation efficiently. Still other compression techniques may bedeveloped in the future or are presently under development. For example,a new video compression standard known as HEVC/H.265 is underdevelopment in the JCT-VC committee. The HEVC/H.265 working draft is setout in “Wiegand et. al., “WD3: Working Draft 3 of High-Efficiency VideoCoding, JCTVC-E603” March 2011, henceforth referred to as “WD3” andincorporated herein by reference in its entirety.

A video encoder can receive uncoded video information for processing inany suitable format, which may be a digital format conforming to ITU-R13T 601 (available from the International Telecommunications Union,Place des Nations, 1211 Geneva 20, Switzerland, www.itu.int, and whichis incorporated herein by reference in its entirety) or in some otherdigital format. The uncoded video may be organized both spatially intopixel values arranged in one or more two-dimensional matrices as well astemporally into a series of uncoded pictures, with each uncoded picturecomprising one or more of the above-mentioned two-dimensional matricesof pixel values. Further, each pixel may comprise a number of separatecomponents used to represent color in digital format. One common formatfor uncoded video that is input to a video encoder has, for each groupof four pixels, four luminance samples which contain informationregarding the brightness/lightness or darkness of the pixels, and twochrominance samples which contain color information (e.g., YCrCb 4:2:0).

One function of video encoders is to translate (more generally“transform”) uncoded pictures into a bitstream, packet stream, NAL unitstream, or other suitable transmission format (all referred to as“bitstream” henceforth), with goals such as reducing the amount ofredundancy encoded into the bitstream to thereby increase transmissionrates, increasing the resilience of the bitstream to suppress bit errorsor packet erasures that may occur during transmission (collectivelyknown as “error resilience”), or other application-specific goals.Embodiments of the present invention provide for at least one of theremoval or reduction of redundancy, the increase in error resilience,and implementability of video encoders and/or associated decoders inparallel processing architectures.

One function of video decoders is to receive as its input a coded videoin the form of a bitstream that may have been produced by a videoencoder conforming to the same video compression standard. The videoencoder then translates (more generally “transforms”) the received codedbitstream into uncoded video information that may be displayed, stored,or otherwise handled.

Both video encoders and video decoders may be implemented using hardwareand/or software configurations, including combinations of both hardwareand software. Implementations of either or both may include the use ofprogrammable hardware components such as general purpose centralprocessing units (CPUs), such as found in personal computers (PCs),embedded processors, graphic card processors, digital signal processors(DSPs), field programmable gate arrays (FPGAs), or others. To implementat least parts of the video encoding or decoding, instructions may beneeded, and those instructions may be stored and distributed using oneor more non-transitory computer readable media. Computer readable mediachoices include compact disc read-only memory (CD-ROM), digitalvideodisc read-only memory (DVD-ROM), memory stick, embedded ROM, orothers.

Video compression and decompression refer to certain operationsperformed in a video encoder and/or decoder. A video decoder may performall, or a subset of, the inverse operations of the encoding operations.Unless otherwise noted, techniques of video encoding described hereinare intended also to encompass the inverse of the described videoencoding techniques (namely associated video decoding techniques).

The uncompressed, digital representation of video can be viewed as asample stream, wherein the samples can be processed by the video displayin scan order. One type of boundary often occurring in this samplestream is the boundary between pictures in the sample stream. Many videocompression standards recognize this boundary and often divide the codedbitstream at these boundaries, for example, through the insertion of apicture header or other metadata at the beginning of each uncodedpicture. Other boundaries that may occur in the sample stream includeslice and tile boundaries, which may occur within an uncoded picture, asdescribed below.

Prediction in video coding can occur at many levels.

One level is referred to henceforth as the “entropy coding level” andthe prediction at that level is referred to as “encoding prediction”. Inthis level, the decoding of an entropy coded symbol may require thesuccessful decoding of previous entropy coded symbols. All or nearly allcurrent video compression standards break the encoding prediction atboth the picture and the slice level. That is, at the detection of apicture or slice header in the bitstream (or equivalent), the entropycoding related states used in the entropy coding are reset to aninitialization state. One example for entropy coded prediction is thereset of CABAC states in ITU-T Rec. H.264.

Further, there can be coding mechanisms that do not fall into the commonunderstanding of entropy coding related prediction, as defined above,but which are still related to the reconstruction control informationassociated with the bitstream, rather than pixel values. As an example,even some older standards such as the ITU-T Rec. H.261 standard allowcoding of motion vectors as relative to one or more previously codedmotion vectors. The detection of a group-of-blocks (GOB), slice orpicture header resets this prediction vector to (0, 0).

There are also prediction mechanisms that span multiple pictures. Forexample, motion compensation can use (possibly motion compensated) pixelvalues from one or more reference pictures for prediction. This type ofprediction is broken through the macroblock type (or equivalent). Forexample, intra macroblocks do not generally use prediction fromreference pictures, whereas inter macroblocks may. Intra and Interslices, in this sense, are simply accumulations of macroblocks belongingto those different macroblock types.

There are also prediction levels that include prediction based on pixelvalues that have already been reconstructed during the reconstructionprocess of the picture being encoded. One example is intra predictionmechanisms, such as the ones described in Annex I of ITU-T Rec. H.263.(Similar mechanisms are available in other video coding standards aswell.)

In addition to prediction mechanisms, several video coding standardsspecify filters for performing in-loop filtering. One example is thein-loop filter specified in Annex J of ITU-T Rec. H.263.

For some applications, it may be advantageous to segment the picturebeing encoded into smaller data blocks, which segmenting can occur priorto, or during, the encoding. Two use cases for which picturesegmentation may be advantageous are described below.

The first such use case involves parallel processing. Previously,standard definition video (e.g., 720×480 or 720×576 pixels) was thelargest format in widespread commercial use. More recently HD (up to1920×1080 pixels) formats as well as 4k (4096×2048 pixels), 8k(8192×4096 pixels), and still larger formats are emerging and findinguse in a variety of application spaces. Despite the increase inaffordable computing power over the years, as a result of the very largepicture sizes associated with some of these newer and larger formats, itis often advantageous to leverage the efficiency of parallel processingto the encoding and decoding processes. Parallel encoding and decodingmay occur, for example, at the instruction level (e.g., using SIMD), ina pipeline where several video coding units may be processed atdifferent stages simultaneously, or on a large structure basis wherecollections of video coding sub units are processed by separatecomputing engines as separate entities (e.g., a multi-core generalpurpose processor). The last form of parallel processing can requirepicture segmentation.

The second such use case involves picture segmentation so as to create abitstream suitable for efficient transport over packet networks. Codecswhose coded video is transported over IP and other packet networks canbe subject to a maximum transmission unit (“MTU”) size constraint. It issometimes advantageous for the coded slice size to be such that theresulting packet containing the coded slice is as close to the MTU sizeas possible without exceeding that size, so as to keep thepayload/packetization overhead ratio high, while avoiding fragmentation(and the resulting higher loss probability) by the network.

The MTU size differs widely from network to network. For example, theMTU size of many Internet connections may be set by the smallest MTUsize of network infrastructure commonly used on the Internet, whichoften corresponds to limitations in Ethernet and may be roughly 1500bytes.

The number of bits in a coded picture depends on many factors such asthe source picture's dimensions, the desired quality, the complexity ofthe content in terms of suitability for prediction, and other factors.However, even at moderate quality settings and content complexity, forsequences of HD resolution and above, the size of an average codedpicture easily exceeds the MTU size. For example, a video conferencingencoder can require about 2 MBit/s to encode a 720p60 video sequence.This results in an average coded picture size of roughly 33333 bits or4167 bytes, which is considerably more than the 1500 bytes of theInternet's MTU size. At higher resolutions, the average picture sizeincreases to values significantly above the Internet's MTU size.Assuming a similar compression ratio as in the 720p60 example above, a4096×2048 (4k) video at 60 fps (4kp60) may require over 300,000 bits, or25 MTU-sized packets for each coded video picture.

In many previous video coding standards (for example, up to andincluding WD3), a picture segment (or, at least, one form of a picturesegment) is known as a “slice”. In the following description, any kindof (e.g., video coding based) picture fragmentation that breaks at leastone form of in-picture prediction, in-loop filtering, or other codingmechanism, may be referred to generally as a “slice”. As such,structures such as the Group Of Blocks (“GOB”) in ITU.T Rec. H.261 orITU Rec. H.263 (available from the ITU; see above for H.264), slices inH.264 or the MPEG family of standards, may each constitute a “slice” asthis term is used herein throughout. However, fragmentation units ofRFC3984 or data partitions of H.264 may not constitute a “slice”, asthis term is used herein throughout, because they subdivide thebitstream of a coded picture and do not break in-picture prediction,in-loop filtering or another coding mechanism.

Referring to FIG. 1, shown is an example 100 of picture segmentationusing slices. A picture 101 is broken into two scan order slices 102,103. The slice boundary is shown as a boldface line 104. The firstmacroblock 105 of the second slice 103 has address 11. The correspondingbitstream 106 for transmitting the picture 101, for example, whengenerated using the H.264 standard, can contain one or more parametersets 107 that do not contain information about the slice boundaries,followed by the slice headers 108, 110 and slice data 109,111 of the twoslices 102, 103. The slice header 110 of the second slice 103 is shownenlarged. The dimensions of the uncoded slice 103, for example, aredetermined by a decoder by a combination of at least two factors. First,the slice header 110 contains the address of the first macroblock 105 ofslice 103. Second, the end of the slice is determined, for example, bythe detection of a new slice header in the bitstream or, in the depictedexample, by the end of the coded picture in the bitstream 112, i.e.,after macroblock 24. All macroblocks between the first macroblock andthe end of the slice make up the slice. It is noted that scan ordermodifications, such as Flexible Macroblock Ordering of H.264, can changethe number of macroblocks in the slice by creating gaps.

One advantage of using slices over media-unaware segmentationmechanisms, such as, for example, those provided by IP at the routinglayer, is that slices are at least to a certain extent independentlydecodeable (as discussed below in more detail), by breaking certaintypes of prediction at the boundaries between slices. The loss of oneslice therefore does not necessarily render the other slices of a codedpicture unusable or un-decodeable. Depending on the implementation of afragmentation mechanism, the loss of a fragment, in contrast, may wellrender many other fragments unusable because fragmentation, as this termis used herein throughout, does not break any form(s) of prediction.

WD4 (B. Bross et. al., “WD4: Working Draft 4 of High-Efficiency VideoCoding”, available fromhttp://wftp3.itu.int/av-arch/jctvc-site/2011_07_F_Torinof) is a draftspecification relating to a digital video coding standard indevelopment, which may be referred to as High Efficiency Video Coding(HEVC) or H.265. In addition to slices, WD4 also includes a picturesegmentation mechanism known as “Tiles”. According to WD4, a sourcepicture can be divided into rectangular units called tiles, such thateach pixel of the source picture is part of a tile (other constraintsmay also apply). A tile is, therefore, a rectangular part of a picture.Tile boundaries are determined by coordinates available in high-levelsyntax structures, which are known in WD4 as parameter sets. Tiles aredescribed in more detail below.

With the possible exception of inter picture prediction, each of thein-picture prediction mechanisms or coding mechanisms described abovemay be broken by the decoding of a picture header (or equivalent, suchas the decoding of a slice with a frame number different from theprevious slice). Whether those prediction mechanisms are broken acrossslice or tile boundaries depends on the video compression standard, andthe type of slice in use.

In H.264, slices may be independently decodeable with respect to motionvector prediction, intra prediction, CA-VLC and CABAC states, and otheraspects of the H.264 standard. Only inter picture prediction (includingimport of pixel data outside of the slice boundaries through motioncompensation) is allowed. While this decoding independence increaseserror resilience, disallowing the aforementioned prediction across sliceboundaries reduces coding efficiency.

In H.263, a video encoder has more flexibility in selecting whichprediction mechanisms are broken through the use of slices or GOBs withnon-empty GOB headers. For example, there is a bit included in thepicture header, selectable when Annex R is in use, which signals to thedecoder that no prediction or filtering at all occurs across slice/GOB(with non-empty headers) boundaries. Certain prediction mechanisms, suchas motion vector prediction are broken across GOBs with non-emptyheaders and across slice boundaries, regardless of the state of Annex R.Others are controlled by Annex R. For example, if the bit is not set,motion vectors may point outside the spatial area co-located with thecurrent slice/GOB with non-empty header in the reference picture(s),thereby potentially “importing” sample values that are used for motioncompensation into the current slice from an area that is not inside ofthe geometric area of the slice/GOB in the reference picture. Further,unless Annex R is active, loop filtering may incorporate sample valuesoutside of the slice/GOB. Similarly, there is another bit in the pictureheader that enables or disables Intra prediction.

However, in most standards, the decision of breaking in pictureprediction is made at least at picture granularity, and in some cases atsequence granularity. In other words, using H.263 as an example, it isnot possible to mix slices in a given picture that have the deblockingfilter enabled or disabled (respectively), nor is it possible toenable/disable intra prediction at the slice level.

As already described, picture segmentation allows breaking a pictureinto spatial areas smaller than a whole picture. While the most commonapplications for picture segmentation, as described, appear to be MTUsize matching and parallelization, picture segmentation can also be usedfor many other purposes, including those that adapt the segment size andshape to the content. Region of interest coding is one of severalexamples. In such cases, it is possible that certain parts of a picturecan be more efficiently coded than others (in the sense that spending alower number of bits for encoding yield comparable visual experience)when different coding tools, including different prediction mechanisms,are applied. For example, some content may benefit from deblockingfiltering and may not respond well to intra prediction, whereas othercontent in the same picture may better be coded without deblockingfiltering, but could benefit from intra prediction. A third content maybest be coded with both deblocking filtering and intra predictionenabled. All this content can be located in the same picture when thepicture is tiled, which occurs, for example, in interview situations, orin video conferencing.

One shortcoming of the existing mechanisms for prediction breaking atsegment boundaries is that the enablement and/or disablement of theprediction breaking is generally hard-coded into the existing videocoding standards, thereby making it difficult or impossible toselectively break prediction mechanisms at segment boundaries based, forexample, on the characteristics of the content to be encoded.

A need therefore exists for an improved method and system to enable ordisable, on a per slice basis, prediction and in-loop filteringmechanisms individually, or as a group. Accordingly, a solution thataddresses, at least in part, the above and other shortcomings isdesired.

Further, a need exists on a per picture (or group of pictures,sequences, etc.) basis to enable, or disable prediction mechanismsand/or in-loop filtering mechanisms across header-less (or equivalent)picture segment boundaries (such as tile boundaries) individually, or asa group. Accordingly, a solution that addresses, at least in part, theabove and other shortcomings is desired.

SUMMARY

Embodiments of the present invention provide method(s) and system(s) forencoding and/or decoding video pictures in which a plurality ofprediction and in-loop filtering tools for a picture segment may beselectively enabled or disabled.

According to one aspect of the invention, an encoder may indicate forone or more prediction tools whether that tool may take information fromoutside the picture segment currently being processed as referenceinformation for processing within that picture segment. The encoder mayprovide such indication for a single prediction tool (for example,entropy prediction, intra prediction, motion compensated prediction,motion vector prediction, henceforth referred to as prediction tools)and/or a single filtering tool (for example, adaptive interpolationfiltering, adaptive loop filtering, deblocking, filtering, sampleadaptive offset, henceforth referred to as loop filter tools), as wellas others. Alternatively, the encoder may provide such indication for aplurality of pre-defined tools or a predefined groups of tools that mayinclude any of the above-noted prediction and loop filter tools, as wellas others. Doing so may be useful in support of parallelization ofencoders and decoders as well as for certain application scenarios suchas soft continuous presence (stitching together coded pictures in thecompressed domain).

According to an aspect of the invention, when header-less picturesegmentation (such as tiles) is in use, the encoder may indicate to theprediction tool, loop filter tool, or plurality of pre-defined groups oftools, whether that tool may take information across horizontal,vertical, or both horizontal and vertical tile boundaries to be used asreference information.

In the particular case of H.264 or HEVC, as an example, the encoder mayset values for “coding interrupt indication” flags used for predictionand in-loop filtering tools such as: intra prediction referencing samplevalues outside the slice/tile boundary; vectors referencing samplevalues outside the slice/tile boundary (i.e., through motioncompensation); use of CABAC states outside the slice/tile boundary; useof CA-VLC states outside the slice/tile boundary; use of PIPE or similarV2V entropy coding states outside the slice/tile boundary (HEVC-only);and, use of states and sample values outside the slice/tile boundary ofin-loop filters such as adaptive interpolation filters, adaptive loopfilters, deblocking loop filters, or sample adaptive offset.

According to an aspect of the invention, the use or other enablement ofcoding tools may not be indicated in the form of flags, but throughmanipulation of other data structures, such as “coding interruptindication” integers that may, in some cases, combine a plurality of theaforementioned flags, or preferred permutations of those flags, into asingle symbol.

According to an aspect of the invention, the maximum length of a motionvector pointing outside of a slice boundary may be coded in anappropriate entropy coding representation of an integer, therebyindicating not only the non-use of motion compensation up to thedistance allowed by the level in use, but also the maximum that isallowed, which can, for example, aid resource allocation in a decoderimplementation.

According to an aspect of the invention, at least one of theaforementioned coding interrupt indication flags or other data codinginterrupt indication structures may be stored in the slice header,picture header, parameter set, or equivalent.

According to an aspect to the invention, a decoder may react to thepresence of the flags or other data structures by breaking the indicatedprediction tools across slice/tile boundaries as opposed to otherpotentially suitable boundaries.

In one broad aspect, there is provided a method for decoding a codedvideo picture comprising a plurality of segments. The method mayinclude, in respect of at least one segment of the coded video picturenot having an associated segment header, obtaining from the coded videopicture at least one indication of at least one prediction or in-loopfiltering operation to be applied to the coded video picture, andcontrolling the at least one prediction or in-loop filtering operationin response to the at least one indication. In some cases, the codedvideo picture may contain at least two segments without an associatedsegment header.

In another broad aspect, there is provided a method for encoding a videopicture comprising a plurality of segments. The method may include, inrespect of at least one segment of the video picture not having anassociated segment header, obtaining at least one indication of at leastone prediction or in-loop filtering operation to be applied to the atleast one segment not having an associated segment header, andcontrolling the at least one prediction or in-loop filtering operationduring encoding of the video picture in response to the at least oneindication. In some cases, the video picture may contain at least twosegments without an associated segment header.

In yet another broad aspect, there is provided a non-transitory computerreadable media having computer executable instructions stored thereonfor programming one or more processors to perform a method of decoding acoded video picture comprising a plurality of segments. The method mayinclude, in respect of at least one segment of the coded video picturenot having an associated segment header, obtaining from the coded videopicture at least one indication of at least one prediction or in-loopfiltering operation to be applied to the coded video picture, andcontrolling the at least one prediction or in-loop filtering operationin response to the at least one indication. In some cases, the codedvideo picture may include at least two segments without an associatedsegment header.

In yet another broad aspect, there is provided a non-transitory computerreadable media having computer executable instructions stored thereonfor programming one or more processors to perform a method of encoding avideo picture comprising a plurality of segments. The method mayinclude, in respect of at least one segment of the video picture nothaving an associated segment header, obtaining at least one indicationof at least one prediction or in-loop filtering operation to be appliedto the at least one segment not having an associated segment header, andcontrolling the at least one prediction or in-loop filtering operationduring encoding of the video picture in response to the at least oneindication. In some cases, the video picture contains at least twosegments without an associated segment header.

In yet another broad aspect, there is provided a data processing systemthat includes at least one of a processor and accelerator hardwareconfigured to execute a method of decoding a coded video picturecomprising a plurality of segments. The method may include, in respectof at least one segment of the coded video picture not having anassociated segment header, obtaining from the coded video picture atleast one indication of at least one prediction or in-loop filteringoperation to be applied to the coded video picture, and controlling theat least one prediction or in-loop filtering operation in response tothe at least one indication. In some cases, the coded video picturecontains at least two segments without an associated segment header.

In yet another broad aspect, there is provided a data processing systemthat includes at least one of a processor and accelerator hardwareconfigured to execute a method of encoding video picture comprising aplurality of segments. The method may include, in respect of at leastone segment of the video picture not having an associated segmentheader, obtaining at least one indication of at least one prediction orin-loop filtering operation to be applied to the at least one segmentnot having an associated segment header, and controlling the at leastone prediction or in-loop filtering operation during encoding of thevideo picture in response to the at least one indication. In some cases,the video picture contains at least two segments without an associatedsegment header.

In some embodiments, according to any of the above aspects, the at leastone prediction or in-loop filtering operation may include at least oneof entropy prediction, intra prediction, motion vector prediction,motion compensated prediction, adaptive loop filtering, adaptiveinterpolation filtering, deblocking filtering, or sample adaptiveoffset.

In some embodiments, according to any of the above aspects, the at leastone of a plurality indications may be derived from at least onecombination indications.

In some embodiments, according to any of the above aspects, the at leastone indication may be coded as a vector indicative of the maximum lengthof a motion vector.

In some embodiments, according to any of the above aspects, the at leastone indication may be coded in a parameter set.

In accordance with further aspects of the present invention, there isprovided an apparatus such as a data processing system, a method foradapting this apparatus, as well as articles of manufacture such as anon-transitory computer readable medium or product having programinstructions recorded and/or stored thereon for performing any of themethods described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

Further features and advantages of the embodiments of the presentinvention will become apparent from the following detailed description,taken in combination with the appended drawings, in which:

FIG. 1 is a diagram illustrating an exemplary picture having scan orderslices, and a bitstream representing the coded picture in accordancewith an embodiment of the invention;

FIG. 2 is a diagram illustrating tiles and slices in accordance with anembodiment of the invention;

FIG. 3 is a block diagram illustrating a coded bitstream in accordancewith an embodiment of the invention;

FIG. 4 is a block diagram illustrating a coded bitstream in accordancewith an embodiment of the invention;

FIG. 5 is a block diagram illustrating a coded bitstream in accordancewith an embodiment of the invention;

FIG. 6 is a flow diagram illustrating the operation of an exemplarydecoder in accordance with an embodiment of the invention;

FIG. 7 is a flow diagram illustrating the operation of an exemplarydecoder when decoding a slice in accordance with an embodiment of theinvention; and,

FIG. 8 is a block diagram illustrating a data processing system (e.g., apersonal computer (“PC”)) based implementation in accordance with anembodiment of the invention.

It will be noted that throughout the appended drawings, like featuresare identified by like reference numerals.

DETAILED DESCRIPTION OF EMBODIMENTS

In the following description, details are set forth to provide anunderstanding of the invention. In some instances, certain software,circuits, structures and methods have not been described or shown indetail in order not to obscure the invention. The term “data processingsystem” is used herein to refer to any machine for processing data,including the computer systems, wireless devices, and networkarrangements described herein. Embodiments of the present invention maybe implemented in any computer programming language provided that theoperating system of the data processing system provides the facilitiesthat may support the requirements of these embodiments. Embodiments ofthe present invention may also be implemented in hardware or in acombination of hardware and software.

At least some embodiments of the present invention relate to selectivelybreaking prediction mechanisms and/or selectively disabling in-loopfiltering mechanisms in conjunction with picture segmentation in videocompression.

Terms such as “segment” or “picture segment” are used hereinafter inreference to any one or more macroblocks or equivalent (for example,treeblocks in WD4) that are smaller than the whole picture, and at whoseboundaries at least one form of prediction is broken and/or at least oneform of in-loop filtering is disabled. H.264-like Slices as well asWD4-like tiles, as described below (with tile boundary_independence_idcequal to 1) are non-limiting examples of segments.

FIG. 2 shows an example 200, where a picture 201 is divided into twotiles 202, 203 by a vertical dividing tile boundary 204, depicted as abold, solid line. Tiles can co-exist with slices within the picture 201.For example, the picture 201 is divided into two slices by the sliceboundary 205 at the same time as being divided into the tiles 202, 203by the tile boundary 204. Tiles (with tile_boundary_independence_idcequal to 1), as described in WD4, may in one or more respects generalizeanother type of picture segment, referred to as columns, which isdescribed further in co-pending U.S. patent application Ser. No.13/336,675, filed even date herewith, entitled “METHOD AND SYSTEM FORPICTURE SEGMENTATION USING COLUMNS”, the entire contents of which areincorporated herein by reference.

The bitstream 206 corresponding to transmission of the picture 201 cancontain, for example, parameter sets 207 or other high-layer syntaxelements that include tile boundary information 208 for identifying thetile boundary 204. The parts of the bitstream other than the parametersets 207, however, do not include any information about tilesboundaries. A decoder can identify the tile to which a coded macroblock(also known as Largest Coding Unit (LCU) or (in WD4) treeblock) belongsby associating its internal state information of the macroblockcurrently being processed and the information of the tiles dimensionsknown from the parameter sets 207.

One difference between tiles and other rectangular picture segmentationmechanisms such as rectangular slices (a sub-mode of Annex K of ITU-TRec. H.263) is that a tile (unlike the rectangular slices) does notrequire a header. The physical dimensions of a tile may be definedinstead in the parameter set without inclusion of a header. In certaincases (with tile_boundary_independence_idc equal to 1), tile boundariesaccording to WD4 interrupt all in-picture prediction mechanisms, butallow for reference of samples in reference pictures that are notco-located with samples associated with the tile for which motioncompensation is being performed. In addition, tile boundaries do notinterrupt in-loop filtering, including the deblocking filter, sampleadaptive offset filter, and adaptive loop filter.

However, it may also be convenient or desirable for an encoder ordecoder to break a different set of prediction mechanisms using tiles.For example, at extremely high resolution, it can be advantageous tosplit the video pictures into tiles subject to requirements that motionvectors are not allowed to point outside of tile boundaries, and/or thata tile boundary is treated by encoder and decoder as a picture boundary(akin to H.263 annex R), or similar, thereby, for example, avoiding notonly motion compensation across tile boundaries but also in-loopfiltering.

In other cases, it may be convenient or desirable for the encoder ordecoder to be able to handle full resolution video encoding except forthe entropy coding of symbols. Such an encoder or decoder might, forexample, involve dedicated signal processing hardware for sample basedprocessing, but may use a general purpose multicore CPU for entropycoding and/or decoding, where a single core would not be able to handlethe load (in HEVC, especially CABAC entropy coding is known to becomputationally demanding). Accordingly, in order to support this usecase, entropy coding may need to be broken at tile boundaries, whileother in picture or inter picture prediction mechanisms may be capableof crossing slice and/or tile boundaries.

In still other cases, it may be convenient or desirable for the encoderor decoder to allow for limited cross-processor coordination across tileboundaries. In such cases, references to pixel values may not bepossible, while reference to control information (such as theinformation necessary for motion vector prediction) may be available onthe communication channel between the processors. In such a case, Intraprediction may not be possible, but motion vector prediction could beused.

There can be coding tools that are not directly related to predictionbut still may advantageously be interrupted across slice or tileboundaries. For example, co-pending U.S. patent application Ser. No.13/286,828, filed Nov. 1, 2011, entitled “ADAPTIVE INTERPOLATION INDIGITAL VIDEO CODING”, the entire contents of which are incorporatedherein by reference, discloses an adaptive interpolation filter whosenature and coefficients can be encoder-selectable. It can beadvantageous to limit the use of samples outside of a slice forinterpolation filtering. Similarly, WD4 includes an adaptiveinterpolation filter, whose control is, at least in part, derived fromcertain pixels. It can be advantageous to limit this derivation topixels only within the slice or tile boundary. It can also beadvantageous to limit the filtering itself (in contrast to thederivation of filter control information) to pixels within the slice ortile boundaries. Also, WD4 includes other loop filters such as theadaptive loop filter (concerned with filtering all samples), adeblocking filter (concerned with filtering block boundaries) and afiltering mechanism known as sample-adaptive offset. These filters mayall share properties similar to AIF. For example, in case of theadaptive loop filter, as specified in WD4, it can be advantageous to(possibly independently) disable access to information used to derivethe filter taps across tile boundaries, as well as disable filteringacross tile boundaries itself.

The boundaries of segments may be defined by picture (or higher) levelsyntax structures, such as, for example, parameter sets when using WD4tiles, by segment header information (such as, for example, H.263 AnnexK rectangular slices), by a combination of placement of a segment headerin the bitstream and encoder/decoder state (such as, for example, H.264slices when Flexible Macroblock Ordering (FMO) is not in use), or acombination of two or more of the aforementioned mechanisms (i.e., FMOdefines slice groups, and the picture segment is defined within theslice group through a combination of slice header placements in thebitstream (identifying the first macroblock of the slice by its address)and implicit ascendance of macroblock addresses within the slice groupuntil the end of the slice is detected through bitstream parsing orother means).

Described now are first mechanisms that allow the selection ofprediction tools for tile boundaries, followed by mechanisms that allowthe selection of prediction tools for slice boundaries. Finally,described is the interworking of the two mechanisms.

Referring to an example 300 in FIG. 3, shown is a coded bitstream 301comprising a parameter set 302 and two coded slices 304, 305. The codedslices 304, 305 may belong to one or two coded pictures. In WD4, apicture boundary can be identified by a slice header with an LCU addressof 0. The parameter set 302 may include tile control information 303(such as tile boundaries), and the information in the parameter set 302is, in this example, assumed to pertain to both coded slices (i.e. theparameter set reference in the slice header contains the same index). Inmany WD4 and H.264 based systems, a parameter set pertains to tens,hundreds, or more slices.

According to an embodiment, the parameter set 302 may contain aplurality of prediction tool indication flags (PTIs). When the PTIs areset (i.e., enabled), for example, prediction across segment boundarieswhichever encoding or decoding tool is associated with the flag may beallowed; otherwise when the PTIs are not set (i.e., disabled), suchprediction may be forbidden. Flags may be defined, for example, forentropy coding prediction 306, intra prediction 307, motion vectorprediction 308, motion compensated prediction 309, adaptive loopfiltering 310, adaptive interpolation filtering 311, deblockingfiltering 312, sample adaptive offset 313, and possibly other predictionand in-loop filtering tools defined in the video coding mechanism.

Including PTIs for individual prediction and in-loop filteringmechanisms, that pertain to all slices and pictures that reference theparameter set, may help to tailor the bitstream to the encoding and/ordecoding environment, such as the hardware architecture of the encoderor decoder. Since the flags may be part of a parameter set, which mayapply to many slices or pictures, the overhead of the PTIs in theparameter set may be negligible compared to the benefits they provide.

Referring to the example 400 depicted in FIG. 4, shown is a codedbitstream 401 comprising a parameter set 402, and a coded picturecontaining two slices 403, 404. Each slice starts with a slice header405, 406. Slice header 405 is enlarged to show parts of its information.

According to an embodiment, the slice header 405 may contain a pluralityof prediction tool indication flags (PTIs). When one or more of the PTIsare set, for example, prediction and/or in-loop filtering across segmentboundaries by whichever encoding or decoding tool is associated with theflag may be allowed; otherwise when the PTIs are not set, suchprediction may be forbidden. PTIs may be defined, for example, forentropy prediction 407, intra prediction 408, motion vector prediction409, motion compensated prediction 410, adaptive loop filtering 411,adaptive interpolation filtering 412, deblocking filtering 413, sampleadaptive offset 414, and possibly other prediction and in-loop filteringtools defined in the video coding mechanism.

Including PTIs for individual prediction and in-loop filteringmechanisms that pertain to a given slice, can help to adapt thebitstream to the content, thereby improving coding efficiency.

Described now is how the two mechanisms described above can interact.

Referring to an example 500 shown in FIG. 5, shown is a coded bitstream501 containing a parameter set 502 and two slices 503, 504, eachstarting with a corresponding slice header 505, 506.

Parameter set 502, which is shown enlarged at 507, for example, includestile control information 508 or other information related to header-lesssegment boundaries that, for example, could indicate a vertical tileboundary 204 as shown in FIG. 2. Further, the parameter set 502 maycontain one or more PTIs. Shown here are three PTIs, one associated withentropy prediction 509, one with intra prediction 510, and one withmotion compensation 511. These flags may control the decoder predictionat the tile boundary 204. The tile boundary 204 may, for example, be setby the tile control information 508 such that the picture 201 is dividedvertically into two tiles 202, 203. The mechanism described here mayalso work with other arrangements of tile boundaries, including acombination of vertical and horizontal boundaries.

The coded picture may, for example, also contain two coded slices 503,504, each starting with a corresponding slice header 505, 506. As shownin FIG. 2, the (uncoded) slices corresponding to coded slices 503, 504may, for example, encompass the spatial area of macroblock addresses 1through 14, and 15 through 24, respectively. Slice header 506 is shownenlarged at 512 and may contain a plurality of PTIs. Shown are two PTIs,one associated with intra prediction 513, the other with adaptive loopfiltering (ALF) 514. However, it is noted that there may be, but is norequirement for, an overlap between the PTIs of the parameter set 502 orthe slice header 506.

According to an embodiment, the PTIs 509, 510, 511 of the parameter set502 control the prediction and in-loop filtering across the tileboundary 204 as defined by the tile control information 508.

According to an embodiment, the PTIs 513, 514 of the slice header 512control the prediction and in-loop filtering across the boundariesbetween the slices 503, 504. For example, the slice boundaries of slice504 have one boundary other than the picture boundary, which is markedby the dashed boldface slice boundary line 205.

As a result, in the example 200, some prediction and in-loop filtermechanisms are interrupted by tile boundaries (so to enable distributionof the picture coding effort among several processors), while otherprediction and in-loop filter mechanisms are interrupted selectively,and under control of the slice header 506, at slice boundaries (therebygiving the encoder full control over prediction and in-loop filteringmechanisms being broken, so to be able to select any particularcombination of prediction and in-loop filtering mechanisms for thecontent being coded, included combinations that may be desirable orconvenient for a given application or use).

If a PTI pertaining to the same prediction or in-loop filteringmechanism is present in both parameter set 502 and in a slice header506, and in the situation where the corresponding tile and sliceboundaries are aligned, at least two decoder reactions may be possible.Such choices may be made statically specified in the standard, through aprofile/level selection, or dynamically based on control information ina parameter set or other high layer syntax elements.

One choice is that the PTIs in the parameter set 502 overwritecontradictive information in the slice header 506. This choice may havethe advantage of providing a decoder with certainty that it candistribute segments to various processors or cores, without having toimplement mechanisms that would allow for sharing information betweenthose segments.

Another choice is that the PTIs in the slice header 508 overwritecontradictive information in the parameter set 502. This choice mayallow for greater encoder flexibility in choosing its tools. Still otherreactions may be possible as well.

In order to optimize the encoding of the flags (be they located in sliceheader 508 or parameter set 502), in some cases, it may be beneficial tospecify in a standard any of the following:

(1) some PTIs may not be part of a parameter set or slice header if acertain profile and/or level is indicated, as the prediction or in-loopfiltering tool is not available in this profile/level.

(2) two or more PTIs may be “bundled” into a single combination PTI, if,for example in a certain profile, it is determined that the flexibilityof switching on/off those individual PTIs independently is unnecessaryor even undesirable.

(3) in some cases, a PTI may not best be coded as a Boolean (i.e.binary) parameter. For example, the need for cross processorcoordination in case of motion compensation may be determined at leastin part by the length of the motion vector pointing outside theco-located spatial area covered by the slice or tile. Accordingly, in anembodiment, PTI information may also be coded as an integer or othernon-boolean parameter, thereby to indicate suitable ranges of values forprediction, such as the maximum length of a motion vector pointingoutside of a segment boundary.

(4) in some cases, PTI values may not need to be physically present inthe bitstream, as their values can be derived from other properties ofthe bitstream. For example, an Intra slice may not need to contain a PTIrelated to motion compensation, because motion compensation may, bydesign of the standard, not occur in an intra slice.

Described now is the operation of an encoder that, according toembodiments, may be suitable for use with any of the configurations ofPTI information previously described.

Referring to FIG. 6, in an embodiment, an encoder may operate accordingto flowchart 600. Before encoding a first slice of a video sequence, theencoder may determine (601) the sequence-related settings for PTIs aswell as a tile layout of the video pictures in the video sequence. Thisdetermination may take into account the hardware architecture of theencoder, the hardware architecture of a decoder, a possible tile layoutsuggested or dictated by the hardware architectures, knowledge about atransmission network (if any), such as the MTU size, and so on. In somecases, PTI values may be mandated by a system-level standard, which canbe taken into account in the determination by the encoder. For example,a future digital TV standard may conceivably require that a certain tilelayout and certain PTI settings controlling prediction and in-loopfiltering across tile boundaries be used for certain (high) resolutions,so to enable cost efficient multiprocessor/multicore implementations. Itis possible that only a subset of all PTIs need to be fixed at sequencelevel.

Several options for those settings have already described before.

After the determination, the encoder may encode (602) thesequence-related PTIs into an appropriate high level syntax structure,such as a sequence or picture parameter set, sequence, GOP, or pictureheader. The encoder may also have the option (through the syntaxstructure of the video coding standard) to leave PTI(s) undefined duringthis encoding.

The sequence-related PTIs may stay constant at least for one full videopicture (unless overwritten by slice header based PTIs, as describedlater), but, in many cases, may stay constant for at least one“sequence” (for example, all pictures between two IDR pictures and theleading IDR picture in the video stream), and perhaps during an entirevideo encoding session. For example, the sequence-related PTIs may atleast partly driven by hardware constraints, which are unlikely tochange during a session. Henceforth assumed for convenience is thislatter case.

The encoder continues by encoding slices. In order to do so, the encodercan determine (603) slice-level PTIs, which may interact withsequence-related PTIs, as already described. The slice-level PTIs can beencoded (604) as part of the encoding of the slice header.

The slice can then be encoded (605) according to whichever codingstandard is being applied, such as WD4 or H.264, while taking intoaccount the breaking of prediction and/or in-loop filtering mechanismsacross both slice and tile boundaries as indicated by the PTIs.

The encoding continues (606) with the next slice.

Described now is the operation of a decoder that, according toembodiments, may be suitable for use with any of the configurations ofPTI information previously described.

FIG. 7 is a flowchart 700 of a decoder that may be used in an embodimentof the invention. The decoder may receive (701) a NAL unit from thebitstream and determine its type. If the NAL unit type is indicative ofa parameter set (702), the decoder may perform parameter set parsing andstorage (703) in accordance with the video coding standard employed.(Other high level syntax structures, such as sequence, GOP, or pictureheaders may also be used for this purpose.).

If the NAL unit type is indicative of slice data (704) (other cases notdepicted) the decoder may parse the slice header (705) and then respondin accordance with the information, such as PTI information, codedtherein. For example, the slice header may contain a parameter setreference, and this parameter set may be “activated” (706) as describedin the video coding standard—that is, the values of the parameter setwith the reference indicated become valid. As PTIs may be part of theparameter set, through the activation (706), their values may becomevalid as well.

The slice header may further include its own PTIs, as already described,which may be different from the PTIs included in the parameter set.Options for how to arbitrate between PTI information coded into theslice header and PTI information coded into the parameter set havealready been described. For example, by correlating the slice headerbased PTIs (if present) with the parameter set header PTIs (if present),and taking into account any restrictions that may be present in otherparts of the video coding standards (such as restrictions and/or defaultsettings of PTIs by profiles and levels), the decoder may determine(707) the final PTI settings to be used in decoding the subject slice.It is noted that PTIs may be different for different borders of theslice, depending on the PTI settings of the parameter set and the PTIsettings of the slice header, including the particular case of when aslice boundary is aligned with a tile boundary.

Taking the final PTI settings into account, the decoder may decode (708)the slice using prediction and/or in-loop filtering techniques acrossslice or tile boundaries as indicated by the information coded into thePTIs.

The process continues (709) with the next NAL unit.

Not shown in FIG. 7 is the processing of NAL units other than slice orparameter set NAL units.

FIG. 8 is a block diagram illustrating a data processing system (e.g., apersonal computer (“PC”)) 800 based implementation in accordance with anembodiment of the invention. Up to this point, for convenience, thedescription has not related explicitly to possible physicalimplementations of the encoder and/or decoder in detail. Many differentphysical implementations based on combinations of software and/orcomponents are possible. For example, in some embodiments, the videoencoder(s) and/or decoder(s) may be implemented using custom or gatearray integrated circuits, in many cases, for reasons related to costefficiency and/or power consumption efficiency.

Additionally, software based implementations are possible using generalpurpose processing architectures, an example of which is the dataprocessing system 800. For example, using a personal computer or similardevice (e.g., set-top-box, laptop, mobile device), such animplementation strategy may be possible as described in the following.As shown in FIG. 8, according to the described embodiments, the encoderand/or the decoder for a PC or similar device 800 may be provided in theform of a computer-readable media 801 (e.g., CD-ROM, semiconductor-ROM,memory stick) containing instructions configured to enable a processor802, alone or in combination with accelerator hardware (e.g., graphicsprocessor) 803, in conjunction with memory 804 coupled to the processor802 and/or the accelerator hardware 803 to perform the encoding ordecoding. The processor 802, memory 804, and accelerator hardware 803may be coupled to a bus 805 that can be used to deliver the bitstreamand the uncompressed video to/from the aforementioned devices. Dependingon the application, peripherals for the input/output of the bitstream orthe uncompressed video may be coupled to the bus 805. For example, acamera 806 may be attached through a suitable interface, such as a framegrabber 807 or a USB link 808, to the bus 805 for real-time input ofuncompressed video. A similar interface can be used for uncompressedvideo storage devices such as VTRs. Uncompressed video may be outputthrough a display device such as a computer monitor or a TV screen 809.A DVD RW drive or equivalent (e.g., CD ROM, CD-RW Blue Ray, memorystick) 810 may be used to input and/or output the bitstream. Finally,for real-time transmission over a network 812, a network interface 811can be used to convey the bitstream and/or uncompressed video, dependingon the capacity of the access link to the network 812, and the network812 itself.

According to various embodiments, the above described method(s) may beimplemented by a respective software module. According to otherembodiments, the above described method(s) may be implemented by arespective hardware module. According to still other embodiments, theabove described method(s) may be implemented by a combination ofsoftware and/or hardware modules.

While the embodiments have, for convenience, been described primarilywith reference to an example method, the apparatus discussed above withreference to a data processing system 800 may, according to thedescribed embodiments, be programmed so as to enable the practice of thedescribed method(s). Moreover, an article of manufacture for use with adata processing system 800, such as a pre-recorded storage device orother similar computer readable medium or product including programinstructions recorded thereon, may direct the data processing system 800so as to facilitate the practice of the described method(s). It isunderstood that such apparatus and articles of manufacture, in additionto the described methods, all fall within the scope of the describedembodiments.

In particular, the sequences of instructions which when executed causethe method described herein to be performed by the data processingsystem 800 can be contained in a data carrier product according to oneembodiment of the invention. This data carrier product can be loadedinto and run by the data processing system 800. In addition, thesequences of instructions which when executed cause the method describedherein to be performed by the data processing system 800 can becontained in a computer program or software product according to oneembodiment of the invention. This computer program or software productcan be loaded into and run by the data processing system 800. Moreover,the sequences of instructions which when executed cause the methoddescribed herein to be performed by the data processing system 800 canbe contained in an integrated circuit product (e.g., a hardware moduleor modules) which may include a coprocessor or memory according to oneembodiment of the invention. This integrated circuit product can beinstalled in the data processing system 800.

The above embodiments may contribute to an improved system and methodfor selectively breaking prediction and/or in-loop filtering in videocoding and may provide one or more advantages. For example, includingPTIs for individual prediction and/or in-loop filtering mechanisms, thatpertain to all slices and pictures that reference the parameter set, mayhelp to tailor the bitstream to the encoding and/or decodingenvironment, such as the hardware architecture of the encoder ordecoder. Further, including PTIs for individual prediction and/orin-loop filtering mechanisms that pertain to a given slice, can help toadapt the bitstream to the content, thereby improving coding efficiency.

The embodiments of the invention described herein are intended to beexemplary only. Accordingly, various alterations and/or modifications ofdetail may be made to these embodiments, all of which come within thescope of the invention.

What is claimed is:
 1. A method for decoding a coded video picturecomprising a plurality of segments, the method comprising: in respect ofat least one segment of the coded video picture not having an associatedsegment header, obtaining from the coded video picture at least oneindication of at least one prediction or in-loop filtering operation tobe applied to the coded video picture; and controlling the at least oneprediction or in-loop filtering operation in response to the at leastone indication, wherein the coded video picture contains at least twosegments without an associated segment header.